From Dual to Primal Sub-optimality for Regularized Empirical Risk Minimization

Regularized empirical risk minimization problems are fundamental tasks in machine learning and data analysis. Many successful approaches for solving these problems are based on a dual formulation, which often admits more efficient algorithms. Often, though, the primal solution is needed. In the case of regularized empirical risk minimization, there is a convenient formula for reconstructing an approximate primal solution from the approximate dual solution. However, the question of quantifying the suboptimality of the primal solution so obtained, and how it relates to sub-optimality of the approximate dual solution, has not been well studied. This paper presents two results. First, we show that when the primal has Lipschitz continuous gradient, we can recover an O( )-sub-optimal primal solution from an O( )-sub-optimal dual solution. Second, when the primal is Lipschitz continuous, we can recover an O( √ )-sub-optimal primal solution from an O( )-sub-optimal dual solution. These results imply that an algorithm that is linearly convergent for the dual problem can yield primal iterates that converge R-linearly to the solution of the primal.